Storage of video analysis data for real-time alerting and forensic analysis

ABSTRACT

A method and apparatus for storing video data is provided. Video data, that comprises a series of frames, is received. Information about changes that are detected in the series of frames is generated. The information is aggregated to generate a plurality of video data change records. Each video data change record corresponds to a plurality of frames and includes change information that indicates changes that were detected relative to the corresponding plurality of frames. Events of interest that satisfy specified search criteria are searched for by comparing the specified search criteria against change information in one or more of the plurality of video data change records.

CLAIM OF PRIORITY

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 60/716,729 filed Sep. 12, 2005, the contents of which isincorporated herein in its entirety for all purposes.

FIELD OF THE INVENTION

The present invention relates to efficiently storing video data, andmore specifically, storing information about visual changes that hasbeen aggregated over a series of frames of the video data.

BACKGROUND

Analyzing video streams to determine whether or not any interestingactivities or objects are present is a resource-intensive operation.Software applications are used to analyze video data, attempting torecognize certain activities or objects in the video data. For example,recognition applications exist for recognizing faces, gestures,vehicles, guns, motion, and the like. Often, such applications are usedto analyze surveillance video streams for security purposes.

If a user is interested in whether a particular object (e.g. face orgun) appears in a video stream, a software application may be used todetect the particular object, and store data that records that theobject was detected. Typically, the amount of storage space needed torecord the detection of those objects is relatively small. However,under some circumstances, one may not know ahead-of-time what events ofinterest will occur in a video stream. In such cases, one couldtheoretically try to detect and capture all possible changes that occurwithin the video stream. However, doing so would require a prohibitivelylarge amount of storage space. Not only would storage capacity issuesarise from storing all possible change information, but it would bedifficult to perform searches against such a vast amount of information.

Due to the impracticality of such an all-changes storage technique,current approaches for scanning for suspicious behavior captured invideo necessarily employ human involvement. Not only is significanthuman involvement prohibitively expensive (especially for small tomid-size businesses), people are prone to error. Watching hours of liveor recorded video is extremely fatiguing, which fatigue may result inmissing suspicious activity. Furthermore, a computer may operatecontinuously whereas people require sleep and rest.

Based on the foregoing, there is a need for efficiently storing motionand other change information to reduce the amount of data stored and toincrease search speed.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a flow diagram that illustrates how video data may be stored,according to an embodiment of the invention;

FIG. 2 is a block diagram that illustrates how video data change recordsmay represent varying amounts of video data, according to an embodimentof the invention;

FIG. 3 is a graphical depiction that illustrates how change informationmay be stored on a per-region basis, according to an embodiment of theinvention;

FIG. 4 is a block diagram that illustrates how video data change recordsmay store specific and generalized change information, according to anembodiment of the invention; and

FIG. 5 is a block diagram of a computer system on which embodiments ofthe invention may be implemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

In order to (1) store a minimal amount of video data and (2) quicklysearch video data for certain events, an efficient storage mechanism isproposed. Instead of storing information related to changes detected invideo data on a frame-by-frame basis, the information is aggregatedacross all or most of the corresponding frames and stored as a singlelogical record in a storage system. For example, typical video camerasand display devices operate at approximately 24 video frames per second.If motion is detected within a particular view and the motion lasts forone minute, then instead of storing 1440 different records to representthe motion, the motion information is stored in a single record thatrepresents the 1440 frames corresponding to the motion.

Because the amount of information that is stored is relatively small,the searches for visual changes that satisfy certain criteria can alsobe performed very efficiently. For example, if a user desired to searchfor a certain type of motion in 1440 frames, then only one record wouldhave to be searched, as opposed to 1440 records.

The embodiments of the invention described herein are illustrated in thecontext of video surveillance systems. However, embodiments of theinvention are limited to that context. Embodiments of the invention arealso relevant in other non-surveillance contexts, such as searching forcertain motion patterns in a series of computer-generated frames.

Functional Overview

FIG. 1 is a flow diagram that illustrates how video data may be storedand used to search for changes detected in the video data, according toan embodiment of the invention. In step 102, video data that comprises aseries of frames is received. In step 104, information about visualchanges that are detected in the series of frames is generated. In step106, the generated information is aggregated to generate a plurality ofvideo data change records (VDCRs). Each video data change recordcorresponds to a plurality of frames and includes change informationthat indicates visual changes that were detected relative to thecorresponding plurality of frames.

In step 108, events of interest that satisfy specified search criteriaare searched for by comparing the specified search criteria againstchange information in one or more of the plurality of visual changerecords. As shall be described in greater detail hereafter, changesdetected in the same sequence of video frames may be aggregated atmultiple levels of granularity. For example, in the spatial dimension,the changes may be aggregated at the entire-view level, the quadrantlevel, and the grid-point level. Similarly, in the temporal dimension,the changes may be aggregated at per-week, per-day, per-hour, per-minuteand per-second levels of granularity, or at variable time intervals thatdepend on other criteria.

Definitions

A “video data change record” (VDCR) is a logical composition of one ormore fields, items, attributes, and/or objects. A VDCR corresponds to aplurality of frames and includes change information (described below). AVDCR corresponds to a particular level of temporal and spatialgranularity. A VDCR may contain information about one or more eventsthat were detected in the video data within the spatial/temporal spaceassociated with the VDCR. VDCRs may also store change informationpertaining about other events that do not appear in the frames thatcorrespond to the VDCR. For example, a VDCR may store informationindicating that an audible alarm began ringing during the time intervalassociated with the VDCR, even though there is no indication within thevideo stream of the alarm.

A VDCR may also include, but is not limited to, (a) a start time of whenthe first frame in the plurality of frames was captured, (b) an end timeof when the last frame in the plurality of frames was captured, (c) atime duration indicating the difference between the start time and theend time, (d) type data indicating whether the change corresponds to adetection of motion or only a pixel change, (e) shape data indicating ashape (e.g., person, car) of a moving object that triggered the VDCR,(f) behavior data indicating a behavior (e.g., walking, running,driving) of a moving object that triggered the VDCR, and (g) anindication of whether the VDCR corresponds to an event or a specifiedtime interval.

A VDCR may also contain a reference to the actual video data thatcorresponds to the plurality of frames of the VDCR in order to enable auser of the storage system to view the corresponding video data. If aVDCR contains a start time, then the start time may be used as thereference.

“Change information” is information that indicates visual changes thatare detected in the temporal/spatial interval associated with a VDCR. Inone embodiment, the change information for changes associated with aVDCR is stored in the VDCR. Change information may indicate motion thatis detected in the plurality of frames and/or a change in pixel valuesthat is detected in the plurality of frames, such as brightness and hue.For example, a pixel change may result from the shadow, of a person,that enters and leaves a view represented by the frames. A pixel changemay also result from a light bulb turning on or off that affects thebrightness of objects in the frames. In some instances, the last framein an event may appear as an exact duplicate of the first frame of theevent. For example, suppose a light bulb faded out and then back on. Bysimply differencing the pixel values of the first frame with the pixelvalues of the last frame, the difference may be zero. Thus, the changeinformation may indicate the greatest amount of change. For example, ifthe light bulb mentioned above went out and then back on and thepossible pixel values range from 0-100, the change information mayindicate 100 instead of zero.

Correspondingly, if the change information indicates a motion, then thechange information may further indicate all directions and/or speeds ofthe motion. For example, with a particular view, an object may moveright, left, up, and down. Thus, the change information may indicate alldirections. As another example, if the object moved at five differentspeeds in a certain direction, then change information may indicate thelargest speed.

Any method for detecting and calculating visual changes (whether justpixel change or motion) may be used. Thus, embodiments of the inventionare not limited to any particular method.

Change information may further include information on a per-regionbasis. A “region” is a portion of a two-dimensional view (e.g., capturedby a video camera) of the video data. The view may be divided intomultiple uniform regions, such as in a grid layout. However, a regionmay be of any arbitrary size and shape. Thus, change information mayinclude motion and/or pixel change information for each specified regionof the view for the duration of the plurality of frames that correspondsto the change information.

An “event” is generally associated with a visual change detected invideo data. For example, an event may correspond to the detection of aperson walking in a region of the view. The duration of the event istypically the length of time that the visual change occurs. Once no morevisual change is detected, then the event may end.

An event may be initiated, not only on the detection of visual changeswithin a view, but also upon the occurrence of an external event. Forexample, an event may be triggered by a fire alarm. Once the fire alarmis detected, the frames of video data from that point on are used togenerate a VDCR that represents the event. The event may end, forexample, when the fire alarm ends or when an administrator of a videosurveillance system indicates that the event is completed.

Alternatively, a VDCR may correspond to a specified time intervalinstead of to an event. For example, regardless of whether a visualchange is detected, a VDCR may be generated for each 5-minute intervalafter every hour. As another example, a VDCR may be generated for each24 hour period.

A VDCR may be generated from other VDCRs and not necessarily from thevideo data itself. For example, if a VDCR is generated for each one-hourperiod of each day, then a “day” VDCR may be generated directly from thetwenty-four “hour” VDCRs that correspond to that day. Similarly, a“week” VDCR may be generated from seven “day” VDCRs, and so forth.

Similarly, a view-level VDCR may be generated based on the changeinformation in the corresponding quadrant VDCRs, and the quadrant VDCRsmay be generated based on the change information in the correspondingregion VDCRs.

Storage of Visual Changes

Because a single VDCR may correspond to thousands or millions of frameof video data, the storage space required to store visual changes ismuch smaller than is required otherwise (e.g., storing a VDCR for eachtwo-frame sequence where motion is detected).

FIG. 2 is a block diagram that illustrates how video data change recordsmay represent varying amounts of video data, according to an embodimentof the invention. Video data 201 comprises a series of frames. Eachblock of video data 201 may represent, for example, 100 frames. Thus,VDCR 202 represents 400 frames and VDCR 206 represents 600 frames. VDCRs202-226 are comprised of two sets: a VDCR set 230 and a VDCR set 240.Suppose that each VDCR in VDCR set 230 (i.e., VDCRs 202-214) isgenerated based on events, as opposed to a pre-specified time intervalregardless of visual changes. For example, VDCR 204 represents an eventthat lasted 600 frames, whereas VDCR 206 represents an event that lasted200 frames.

Further suppose that each VDCR in VDCR set 240 (i.e., VDCRs 222-226) isgenerated for each hour. Thus, for example, VDCR 222 may represent hour# 1, VDCR 224 may represent hour #2, and VDCR 226 may represent hour #3.Therefore, a specific period of time may be represented by multipleVDCRs that each represents different periods of time.

In one embodiment, if a VDCR represents a number of frames that is alsorepresented by another VDCR, then each VDCR contains a reference to theother VDCR, as illustrated. Thus, VDCR 222 contains references to VDCRs202 and 204 and VDCRs 202 and 204 each contain a reference to VDCR 222.Similarly, VDCR 226 contains references to VDCRs 210-214 and VDCRs210-214 each contain a reference to VDCR 226.

VDCRs may be stored on disk in specified tables. Each table maycorrespond to VDCRs of a certain type. For example, each table of aplurality of tables may comprise VDCRs that correspond to a specifiedtime interval (e.g., day table, week table, month table, etc.). Asanother example, each table of a plurality of tables may comprise VDCRsthat correspond to certain time frames (e.g., all VDCRs generated inJanuary, 2006, or all VDCRs generated during week #51, etc.).Embodiments of the invention are not limited to how VDCRs are organizedon disk.

Regions

As described above, a two-dimensional view of video data may be dividedinto multiple regions. A region may or not be convex. Multiple regionswithin a view may be of different sizes and shapes.

If a VDCR is generated based on a detected change in pixel values thatis not associated with motion of an object, then the change informationof a particular region may indicate the amount of change in pixel valueswithin that particular region.

If a VDCR is generated based on a motion event, then the changeinformation of a particular region may indicate the direction and/orvelocity of the detected motion within that particular region. Forexample, suppose a ball was thrown through the view of a camera. A VDCRwas generated for the few frames that captured the event. The changeinformation in every region of the view through which the ball traveledwill indicate that motion was detected in that region and may indicatethe direction of the ball and the velocity of the ball.

FIG. 3 is a graphical depiction that illustrates how change informationmay be stored on a per-region basis, according to an embodiment of theinvention. In this example, a camera view is divided into multiple,non-overlapping rectangle regions. Each region in FIG. 3 indicates oneor more directions of motion. For example, region 302 indicates that atleast four directions of a motion have been detected for that particularregion. As this example further illustrates, not every region mustspecify a direction and/or speed, such as region 304. In such a case,the change information for such a region may be empty or include someindicator indicating zero direction and/or speed.

Abstraction of Change Information

In one embodiment, change information may be kept, not only inper-region VDCRs, but also in multi-region VDCRs. For example, if thereare 100 regions within a view, and change information is maintained foreach region for a given region-level VDCR, then view-level VDCRs mayindicate change information for the entire view, quadrant-level VDCRsmay indicate change information for each quadrant of the view, etc. Ineach VDCR, the change information is abstracted to the level of theVDCR. Thus, change information for a view-level VDCR may contain1/00^(th) the information of the information contained in thecorresponding 100 region-level VDCRs. Similarly, the change informationfor a single quadrant-level VDCR may contain 1/25^(th) the informationof the information contained in the corresponding twenty-fiveregion-level VDCRs that correspond to the quadrant.

It should be noted that VDCRs are logical pieces of information, and donot necessarily correspond to distinct records within a repository. Forexample, a single record or data structure within a repository mayinclude a view-level VDCR, its corresponding quadrant-level VDCRs, andits corresponding region-level VDCRs. Similarly, a single record or datastructure may be used to store change information aggregated at the weeklevel, the day, level, the hour level and the minute level. Datastructures that store change data that has been aggregated at multiplelevels of granularity are referred to herein as composite VDCRs.

The nature of the repository used to store VDCRs may vary fromimplementation to implementation. The techniques described herein arenot limited to any particular type of repository. For example, the VDCRsmay be stores in a multi-dimensional database, a relational database, oran object-relational database. In one embodiment, separate relationaltables are used to store VDCRs at different levels of granularity. Thus,one relational table may have rows that correspond to view-level VDCRs,while another relational table may have rows that correspond toregion-level VDCRs. In such an embodiment, indexes may be used toefficiently locate the region-level VDCRs that correspond to aparticular view-level VDCR.

FIG. 4 is a block diagram that illustrates how composite video datachange records may store specific and generalized change information,according to an embodiment of the invention. Video data 401 comprises aseries of frames. Each block of video data 401 may represent any numberof frames, such as one hundred. Thus, composite VDCR 402 may representfour hundred frames. The change information of VDCR 402 may berepresented on a per-region basis, a per-quadrant basis, a per-viewbasis, and/or any other basis. In this example, VDCR 402 comprises viewdata 403, quadrant data 404, and region data 405. VDCR 402 may compriseother information such as whether any visual change was detected in theframes that correspond to VDCR 402 and the type of visual change (e.g.,pixel change, motion).

Alerts

In one embodiment, search criteria may be specified in which allincoming video data is analyzed to determine whether any detected visualchanges satisfy the search criteria. Thus, for an ongoing event (i.e.,before a VDCR is generated for the event), an alert may be triggered onthe precise frame that the change information (accumulated thus far)first satisfied the search criteria. Once the event has completed, theaccumulated change information is stored (in the manner described above)in a VDCR so that future searches with similar search criteria mayreturn that VDCR.

Filters

In one embodiment, several levels of filtering may be performed forvarious reasons, some of which may include (1) reducing noise that maybe generated when generating change information and (2) determiningdominant motion areas and velocities within a scene. The changeinformation of a particular VDCR may be filtered across adjacent regionswithin a frame, or across frames, of the corresponding plurality offrames and adjacent frames within the corresponding plurality of frames,using various methods that include, but are not limited to, smoothingfilters, median filters, and multi-dimensional clustering algorithms.

Searching

With the storage techniques described above, the generated VDCRsfacilitate fast searches across multiple events and specified timeintervals. Because searches are executed against change information, asdescribed above (which may be thought of as meta-data about visualchanges), rather than the entire video data itself, the searches may beperformed much faster than if a user was required to search through theentire video data or search on a frame-by-frame basis of each detectedchange.

Thus, a user may specify search criteria that are compared against eachVDCR. For example, a user may search for all VDCRs that indicate anymotion, where the motion is more than 20 mph. As another example, a usermay search for one VDCR that indicates a pixel change in the lower leftquadrant of a 50% change in brightness.

Multiple indexes may be generated in order to facilitate fastersearches. Such indexes may be based on time, the type of visual change,the speed of a motion, the direction of a motion, etc.

Furthermore, the manner in which change information is stored and thevarying types of information that a VDCR may include make possible manytypes of searches. For example, search criteria of a particular searchmay include (1) multiple ranges of time, (2) the speed of motion in someregions, (3) the direction of motion in other regions, (4) an amount ofpixel change in still other regions, (5) the shape and type of behaviorof multiple detected objects, etc. The number of possible searchcriteria is immeasurable.

Multi-Level Searching—Regions

As described above, change information that is generated from video datamay be aggregated at different levels of spatial granularity. Forexample, the change information stored for a particular time period mayinclude (1) view-level VDCRs that indicate change information relativeto the entire view, (2) quadrant-level VDCRs that indicate changeinformation for each of four quadrants of the view, and (3) region-levelVDCRs that indicate change information for each of a thousand gridregions within the view. The search mechanism may make use of thesedifferent levels of granularity to improve search performance.

For example, suppose a view is divided into one hundred non-overlappingregions. Further, suppose that a user is searching for motion eventsthat occurred over a particular week, and that a million region-levelVDCRs have been generated for each region during that week. Suppose thatthe search criteria includes that a specified type of motion occurredwithin each region of twenty-four specified regions of the view. In thisexample, if the entire search is performed at the region-level ofgranularity, then twenty-four million region-level VDCRs will have to beinspected during the search.

Instead of performing the entire search at the region-level ofgranularity, a multi-level search may be performed. Specifically, duringthe first phase of the multi-level search, each of a million view-levelVDCRs may be inspected to find those view-level VDCRs that indicate thatthe specified motion occurred anywhere within the view. Thedetermination may be based on view-level change information in eachview-level VDCR. The view-level change information of a view-level VDCRindicates whether motion was detected anywhere in the entire view duringthe frames associated with the view-level VDCR. In the present example,the first-level search will involve one million comparisons (one foreach view-level VDCR). For the purpose of explanation, assume that50,000 view-level VDCRs matched the first-level search.

During the second-phase of the multi-level search, quadrant-level VDCRsare inspected. However, rather than inspecting all 4 million of thequadrant-level VDCRs, only the quadrant-level VDCRs that correspond tothe 50,000 view-level VDCRs are searched in the second-level search.Further, if the 24 regions specified in the search criteria only fallwithin two of the four quadrants, then the second-level search need onlyinvolve the quadrant-level VDCRs associated with those two quadrants.Thus, the second phase of the search will involve no more than 100,000quadrant-level VDCRs.

Each quadrant-level VDCR includes quadrant-level data that indicateswhether motion was detected in any portion of the correspondingquadrant. For the purpose of explanation, assume that, based on thequadrant-level VDCRs, only 10,000 view-level VDCRs of the 50,000 VDCRsincluded motion in those two quadrants.

In the third level search, a region-level search is performed againstthe region-level VDCRs that correspond to the 10,000 view-level VDCRs.When searching at the region-level of granularity, 24 region-level VDCRsmay need to be inspected for each of the 10,000 view-level VDCRs.However, because the candidate set of view-level VDCRs has been pruneddown during the first two search phases, the number of region-levelcomparisons performed during the third-level search (240,000, in thepresent example) will typically be far fewer than the number ofcomparisons (24 million) that would have been performed if all searchingwas done at the region-level of granularity.

Multi-Level Searching—Time

As with areas of a view, a search may be separated into a multi-levelsearch according to time. For example, suppose a user wants to findmotion events that occurred between the hours of 1:00 AM and 5:00 AMduring the past week. Further suppose that an hour-level VDCR exists foreach hour and each day. Thus, in the first search level, each day-levelVDCR of the past week is examined to determine whether motion wasdetected in the corresponding day. In the second search level, eachhour-level VDCR that is associated with a day-level VDCR that wasidentified in the first search level is examined to determine whethermotion was detected in the corresponding hour.

In one embodiment, one level of a multi-level search may be performedbased on time and another level of the multi-level search may beperformed based on areas of the view. For example, suppose searchcriteria specifies motion that occurred within a certain area of a viewbetween the hours of 1:00 AM and 5:00 AM during the past week. Thus, thefirst two levels of the search may be used to identify allhour-level/view-level VDCRs of the past week between 1:00 AM and 5:00AM. Subsequent levels of the search may be used to identify allhour-level/region-level VDCRs with change information that indicates thespecified motion in the specified area.

In one embodiment, users may specify the search criteria for each levelof a multi-level search. In another embodiment, multi-level searches maybe performed automatically transparent to the user, beginning atrelatively coarser temporal/spatial granularities and ending at thelevel of granularities of the search criteria that was specified by theuser. Thus, a single set of search criteria may be automatically divided(e.g., by a query compiler) into one or more general searches and onespecific search. Any mechanism for dividing search criteria into amulti-level query may be used. Embodiments of the invention are notlimited to any specific mechanism.

Hardware Overview

FIG. 5 is a block diagram that illustrates a computer system 500 uponwhich an embodiment of the invention may be implemented. Computer system500 includes a bus 502 or other communication mechanism forcommunicating information, and a processor 504 coupled with bus 502 forprocessing information. Computer system 500 also includes a main memory506, such as a random access memory (RAM) or other dynamic storagedevice, coupled to bus 502 for storing information and instructions tobe executed by processor 504. Main memory 506 also may be used forstoring temporary variables or other intermediate information duringexecution of instructions to be executed by processor 504. Computersystem 500 further includes a read only memory (ROM) 508 or other staticstorage device coupled to bus 502 for storing static information andinstructions for processor 504. A storage device 510, such as a magneticdisk or optical disk, is provided and coupled to bus 502 for storinginformation and instructions.

Computer system 500 may be coupled via bus 502 to a display 512, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 514, including alphanumeric and other keys, is coupledto bus 502 for communicating information and command selections toprocessor 504. Another type of user input device is cursor control 516,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 504 and forcontrolling cursor movement on display 512. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

The invention is related to the use of computer system 500 forimplementing the techniques described herein. According to oneembodiment of the invention, those techniques are performed by computersystem 500 in response to processor 504 executing one or more sequencesof one or more instructions contained in main memory 506. Suchinstructions may be read into main memory 506 from anothermachine-readable medium, such as storage device 510. Execution of thesequences of instructions contained in main memory 506 causes processor504 to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

The term “machine-readable medium” as used herein refers to any mediumthat participates in providing data that causes a machine to operationin a specific fashion. In an embodiment implemented using computersystem 500, various machine-readable media are involved, for example, inproviding instructions to processor 504 for execution. Such a medium maytake many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, optical or magnetic disks, such as storage device 510. Volatilemedia includes dynamic memory, such as main memory 506. Transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that comprise bus 502. Transmission media can also take theform of acoustic or light waves, such as those generated duringradio-wave and infra-red data communications. All such media must betangible to enable the instructions carried by the media to be detectedby a physical mechanism that reads the instructions into a machine.

Common forms of machine-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CD-ROM, any other optical medium, punchcards, papertape, anyother physical medium with patterns of holes, a RAM, a PROM, and EPROM,a FLASH-EPROM, any other memory chip or cartridge, a carrier wave asdescribed hereinafter, or any other medium from which a computer canread.

Various forms of machine-readable media may be involved in carrying oneor more sequences of one or more instructions to processor 504 forexecution. For example, the instructions may initially be carried on amagnetic disk of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 500 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 502. Bus 502 carries the data tomain memory 506, from which processor 504 retrieves and executes theinstructions. The instructions received by main memory 506 mayoptionally be stored on storage device 510 either before or afterexecution by processor 504.

Computer system 500 also includes a communication interface 518 coupledto bus 502. Communication interface 518 provides a two-way datacommunication coupling to a network link 520 that is connected to alocal network 522. For example, communication interface 518 may be anintegrated services digital network (ISDN) card or a modem to provide adata communication connection to a corresponding type of telephone line.As another example, communication interface 518 may be a local areanetwork (LAN) card to provide a data communication connection to acompatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 518 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 520 typically provides data communication through one ormore networks to other data devices. For example, network link 520 mayprovide a connection through local network 522 to a host computer 524 orto data equipment operated by an Internet Service Provider (ISP) 526.ISP 526 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 528. Local network 522 and Internet 528 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 520and through communication interface 518, which carry the digital data toand from computer system 500, are exemplary forms of carrier wavestransporting the information.

Computer system 500 can send messages and receive data, includingprogram code, through the network(s), network link 520 and communicationinterface 518. In the Internet example, a server 530 might transmit arequested code for an application program through Internet 528, ISP 526,local network 522 and communication interface 518.

The received code may be executed by processor 504 as it is received,and/or stored in storage device 510, or other non-volatile storage forlater execution. In this manner, computer system 500 may obtainapplication code in the form of a carrier wave.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1. A machine-implemented method, comprising: generating informationabout changes that are detected in video data that includes a series offrames; aggregating the information to generate a plurality of videodata change records, wherein each video data change record correspondsto a plurality of frames and includes change information that indicateschanges that were detected in video data relative to the correspondingplurality of frames; and storing said aggregated video data changerecords for subsequent forensic analysis.
 2. The method of claim 1,wherein: the plurality of video data change records includes aparticular video data change record that corresponds to an eventdetected within the video data; and the change information contained inthe particular video data change record indicates changes that occurredin the frames that correspond to said event.
 3. The method of claim 1,wherein: the plurality of video data change records includes a set ofvideo data change records that each corresponds to a specified timeinterval; and the change information contained in each video data changerecord of the set of video data change records indicates changes thatoccurred in the frames that correspond to the corresponding specifiedtime interval.
 4. The method of claim 1, wherein the change informationcontained in each video data change record of the plurality of videodata change records includes change type information that indicates atleast one of: (a) a change in pixel values that occurred during theplurality of frames that correspond to said each video data changerecord, and (b) a detection of motion of one or more objects in theplurality of frames that correspond to said each video data changerecord.
 5. The method of claim 4, wherein: the change type informationof each video data change record indicates the detection of motion ofone or more objects captured in the plurality of frames that correspondto said each video data change record; and said each video data changerecord further indicates at least one of a direction and a speed of saidmotion.
 6. The method of claim 1, wherein: said plurality of video datachange records includes a plurality of region change records; and eachregion change record of the plurality of region change recordscorresponds to a separate region of a frame and contains region changeinformation about the changes that were detected in the separate regionrelative to the plurality of frames that correspond to the particularvideo data change record.
 7. The method of claim 6, wherein: the regionchange information of said each region change record is filtered basedon other region change information of other region change records of theplurality of region change records; and the other region change recordscorrespond to other regions, of the frame, that are adjacent to theregion that corresponds to said each region change record.
 8. The methodof claim 1, wherein the change information of a particular video datachange record of the plurality of video data change records is filtered,based on the plurality of frames that correspond to said particularvideo data change record.
 9. The method of claim 1, further comprisingsearching for events of interest that satisfy specified search criteriaby comparing the specified search criteria against change information inone or more of said plurality of video data change records.
 10. Themethod of claim 6, wherein: the method further comprises searching forevents of interest that satisfy specified search criteria by comparingthe specified search criteria against change information in one or moreof said plurality of video data change records; said specified searchcriteria indicates a first region of a set of frames in which the eventsof interest must have occurred; searching for the events of interestthat satisfy the specified search criteria includes: identifying a firstset of video data change records that correspond to an area of the viewthat includes said first region and is larger than said first region;and based on the first set of video data change records, identifying asecond set of video data change records that correspond to said firstregion, wherein the change information of each video data change recordin the second set satisfies the specified search criteria.
 11. Themethod of claim 1, wherein: the method further comprises searching forevents of interest that satisfy specified search criteria by comparingthe specified search criteria against change information in one or moreof said plurality of video data change records; said specified searchcriteria specifies time at which the events of interest must haveoccurred; searching for the events of interest that satisfy thespecified search criteria includes: identifying a first set of videodata change records that correspond to a first time interval thatincludes said specified time; and based on said first set of video datachange records, identifying a second set of video data change recordsthat aggregate change information at a finer level of granularity thanthe video data change records in said first set of video data changerecords; wherein the change information of each video data change recordin the second set includes said specified time and satisfies thespecified search criteria.
 12. The method of claim 1, wherein at leastone video data change record of said plurality of video data changerecords corresponds to a time interval that was established, at least inpart, by an event that is not reflected in said video data.
 13. Themethod of claim 9, further comprising: receiving second specified searchcriteria; detecting an event in said video data as the video data isbeing received; generating aggregated change information associated withsaid event; before a video data change record is generated for saidevent, comparing said second specified search criteria against saidaggregated change information; and if said aggregated change informationsatisfies said second specified search criteria, immediately generatingan alert that indicates that said second specified search criteria hasbeen satisfied.
 14. A machine-implemented method, comprising: generatinginformation about changes that are detected in video data that includesa series of frames; and aggregating the information to generate a firstplurality of video data change records, wherein each video data changerecord in the first plurality of change records corresponds to aplurality of frames and includes change information, aggregated at arelatively fine level of granularity, that indicates changes that weredetected relative to the corresponding plurality of frames; andaggregating the information to generate a second plurality of video datachange records, wherein each video data change record in the secondplurality of change records corresponds to a plurality of frames andincludes change information, aggregated at a relatively coarse level ofgranularity, that indicates changes that were detected relative to thecorresponding plurality of frames.
 15. The method of claim 14 wherein:each of the first plurality of video data change records corresponds toregions of a view; and each of the second plurality of video data changerecords corresponds to the entire view.
 16. The method of claim 14wherein: each of the first plurality of video data change recordscorresponds to time intervals of a first duration; and each of thesecond plurality of video data change records corresponds to timeintervals of a second duration, wherein the second duration is longerthan said first duration.
 17. A computer-readable medium carrying one ormore sequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 1. 18. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 2. 19. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 3. 20. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 4. 21. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 5. 22. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 6. 23. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 7. 24. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 8. 25. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 9. 26. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 10. 27. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 11. 28. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 12. 29. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 13. 30. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 14. 31. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim
 15. 32. A computer-readable medium carrying one or moresequences of instructions which, when executed by one or moreprocessors, causes the one or more processors to perform the methodrecited in claim 16.